[Nexthop] don't crash sw_agent in handlePendingUpdates when HwSwitch connections are lost - #1544

Stacked on #1492. The first three commits belong to #1492 ("Fix agent
EventBase double-drive abort on graceful exit") and are carried here only so this
change compiles and tests standalone — it depends on the
SwSwitch::requestGracefulShutdown() API and the
HwSwitchConnectionStatusTable::disconnected() change that PR introduces. Please
review only the last commit; I will rebase to drop the duplicates once #1492 merges.

Summary

fboss_sw_agent crashes with SIGABRT when the hw agent is restarted ~1–2s
after the sw agent starts — i.e. while a fresh sw agent (coldboot) is still
applying its initial pending updates. Under config load this reproduces
every time. The same fatal also fingerprints on mid-test hw agent restarts.

Crashing stack: SwSwitch::handlePendingUpdates →
LOG(FATAL) "Failed to apply update to HW and the update is not marked for HW failure protection".

Root cause

A shutdown/update race in split-agent (multi_switch) mode:

1. The hw agent restart kills the oper delta stream; the in-flight update
   either times out waiting for its ack or hits the DISCONNECTED sync state,
   and MultiSwitchHwSwitchHandler::stateChanged returns
   HWSWITCH_STATE_UPDATE_CANCELLED (nothing applied).
2. HwSwitchConnectionStatusTable::disconnected() sees the last connection
   drop, creates cold boot markers, and calls requestGracefulShutdown() —
   but that only schedules teardown on another event base; the EXITING run
   state is set later, inside stop().
3. Back on the update thread, applied != desired. The only escape hatches are
   isExiting() (still false — the race) and hwFailureProtected() (false
   for initial config/route updates), so it falls into the FATAL.

The CANCELLED status — "connection lost, shutdown imminent", deliberately
distinct from FAILED — is dropped in MultiHwSwitchHandler::stateChanged,
so SwSwitch can't tell a connection loss from a real programming failure.

Fix

Treat "all HwSwitch connections lost" the same as "already exiting" at the
FATAL site:

• HwSwitchConnectionStatusTable::hasActiveConnections() — new accessor.
• MultiHwSwitchHandler::hasActiveHwSwitchConnections() — always true in
  monolithic mode (mirrors isHwSwitchConnected), else consults the table.
• SwSwitch::handlePendingUpdates: if the update failed and no HwSwitch
  connections remain, log an error, increment a new
  hwswitch_disconnected_update_drop counter so the drop is alertable
  (the sole evidence was otherwise one ERR line per dropped update), request
  the graceful shutdown, and fall through like the isExiting() branch so
  blocking callers complete.

The requestGracefulShutdown() here is belt-and-braces: every path that
empties the connection table already requests one (and call_once collapses
the requests), but the drop path must not depend on that invariant — dropping
updates with no pending teardown would leave a zombie agent acking updates
that never touched hardware. A comment at the call site records this so it
doesn't read as dead code.

The connection-table check is a proxy for the dropped CANCELLED status. It is
correct because both cancellation paths erase the table entry before
stateChanged returns to the update thread: the stream-disconnect path via
notifyHwSwitchDisconnected, and the ack-timeout path via the
disconnected() call inside MultiSwitchHwSwitchHandler::waitForOperSyncAck.
A TODO at the site documents the exact alternative — propagating the aggregate
update status out of MultiHwSwitchHandler::stateChanged — which would not
depend on that ordering and would also cover partial cancellation in
multi-HwSwitch topologies (one of several HwSwitches cancelled), a pre-existing
gap not addressed here.

Behavior is unchanged for genuine HW programming failures with a live
connection (still FATAL), HW-failure-protected updates (still throw
FbossHwUpdateError to the caller), and monolithic mode.

Test Plan

New unit test SwSwitchHandlerTest.updateFailureWithNoActiveHwConnectionsDoesNotCrash
in fboss/agent/test/SwitchHandlerTest.cpp covers the dropped update, the
counter bump, and the graceful-shutdown request. This change also adds a
switch_handler_test cmake target so SwitchHandlerTest.cpp is built and run
by the OSS cmake build, which it was not before.

Verified with the cmake build on this exact tree: builds clean, and the unit
test suite passes 1947/1947 including the new test.

The agent-coldboot system test that previously crashed reliably passes with
this change.